Papers with pretraining large language models
Unsupervised Improvement of Factual Knowledge in Language Models (2023.eacl-main)
Copied to clipboard
| Challenge: | Masked language modeling (MLM) is often dominated by high-frequency words that are sub-optimal for learning factual knowledge. |
| Approach: | They propose an approach that forces the model to prioritize informative words in a fully unsupervised way. |
| Outcome: | The proposed approach significantly improves the performance of pretrained language models on factual recall, question answering, sentiment analysis, and natural language inference in a closed-book setting. |
InforMask: Unsupervised Informative Masking for Language Model Pretraining (2022.emnlp-main)
Copied to clipboard
| Challenge: | Masked language modeling is used for pretraining large language models for knowledge-intensive tasks. |
| Approach: | They propose an unsupervised masking strategy that exploits Pointwise Mutual Information to select the most informative tokens to mask. |
| Outcome: | The proposed strategy outperforms random masking and previously proposed masking strategies on the factual recall benchmark LAMA and the question answering benchmark SQuAD v1 and v2. |
Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models (2025.emnlp-main)
Copied to clipboard
Mehdi Ali, Manuel Brack, Max Lübbering, Elias Wendt, Abbas Goher Khan, Richard Rutmann, Alex Jude, Maurice Kraus, Alexander Arno Weber, Felix Stollenwerk, David Kaczér, Florian Mai, Lucie Flek, Rafet Sifa, Nicolas Flores-Herr, Joachim Koehler, Patrick Schramowski, Michael Fromm, Kristian Kersting
| Challenge: | Existing open-source multilingual datasets rely on heuristic filtering methods restricting both their cross-lingual transferability and scalability. |
| Approach: | They propose a systematic approach that curates diverse and high-quality multilingual data at scale while significantly reducing computational demands. |
| Outcome: | Evaluated empirically across 35 languages, the proposed approach outperforms current heuristic filtering methods like Fineweb2 and improves model training quality and retention rates. |
Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large and sparse feed-forward layers (S-FFN) have proven effective in scaling up the model size for pretraining large language models. |
| Approach: | They compare S-FFN architectures for language modeling and compare their performance and efficiency . they found a simpler selection method that selects blocks through their mean aggregated hidden states . |
| Outcome: | The proposed model size and selection method achieve lower perplexity in language model pretraining compared to existing MoE architectures. |